Behind the Scenes: The MPE Spool Queue

During the course of troubleshooting a recent system problem, we observed that the system was extraordinarily slow and unresponsive for a long period of time after MPE was restarted. The cause – an unusually large number of spool files in the queue. Why would this have such a negative affect on performance?

The reason is fairly simple. There are two components that make up the MPE spool queue. The first is the spool files – permanent disk files named Onnnn.OUT.HPSPOOL. The second piece is the Spoolfile Directory (SPFDIR,) which keeps track of all cataloged spool files. It contains all of the information required by the LISTSPF command such as spool file name (not the DFID number,) job name, creator, number of copies, etc. Merely creating a file in OUT.HPSPOOL does not create a spoolfile, it must be cataloged in the Spoolfile Directory.

The SPFDIR is a memory resident table. Consequently, it does not survive a system reboot and must be rebuilt from scratch. You may have even seenthis message on the console during a system reboot, indicating when the process is complete.

The system has finished rebuilding the output spoolfile directory.”

Where does the performance implication come into play? One of the pieces of information that is stored in the SPFDIR for each spool file is the number of lines. Because spool files are variable record files, obtaining the number of lines requires that the entire file be read, record by record, calculating the offset from one record to the next based on each record’s length.

On most systems, with only a few hundred spool files in the queue, the overhead incurred while rebuilding the SPFDIR is rarely noticed. On this particular system, there were over 40,000 spool files consuming almost 4GB of space, and the impact was more than a little noticeable.

There are additional reasons it is recommended that the spool queue not be allowed to get this large. The spool queue was never intended for long term storage of data. A bloated SPFDIR will negatively affect performance of LISTSPF and SPOOLF commands, as they have to scan the directory to qualify spool files. Backups will take longer, as will restores. There is also a limit of ~50,000 spool files before the SPFDIR table is full. A full SPFDIR table will prevent the creation of new spool files, including $STDLIST for jobs, which in turn will prevent jobs from logging on.
Create a job to delete old spool files with DATEOFF.

You can create a maintenance job that runs on a regular basis to clean up old spool files. The task can be incorporated into a job scheduler or the commands could simply be added into a weekly full backup. The trick to automating any job of this type is to be able to calculate backwards a specific number of days from today. We have a freeware program to accomplish this task. Feel free to download a copy of DATEOFF from our website at http://www.beechglen.com/mpe/downloads/.

DATEOFF sets a CI variable named HPDATEOFFSET in the format YYYYMMDD, calculated from an input date and a number of days, which can be plus or minus. Any date can be passed to DATEOFF for the starting point. In this script, the current system date will be passed by using the predefined CI variable !HPYYYYMMDD with a –21 offset to compute a date three weeks in the past. The resulting date is then fed into the SPOOLF command.


# Delete all spool files from the system that are more than
# 21 days old
#
dateoff !hpyyyymmdd,-21
setvar mm,str("!hpdateoffset",5,2)
setvar dd,str("!hpdateoffset",7,2)
spoolf @;seleq=[ date < !mm/!dd/!yyyy ];delete

Ongoing Care and Maintenance

What are the best ways to maintain the spool queue? First, establish policies for how long spool files can remain in the queue before they are either printed or deleted. Create a weekly routine that automatically deletes spool files that have exceeded the agreed upon time frame as in the example above. Consider archiving spool files using one of the third party packages that perform this task such as ESPUL from RAC (http://www.racc.com), VISTA from Quest (http://www.quest.com), or perhaps consider creating one of your own. Spool files can be converted to HTML documents using SF2HTML, freely available from http://www.beechglen.com/mpe/downloads/. The resulting HTML documents can be downloaded to a workstation or server, and then burned to CD.